Method, system, and medium having stored thereon instructions that cause a processor to execute a method for obtaining image information of an organism comprising a set of optical data

ABSTRACT

The present disclosure relates to methods and systems for obtaining image information of an organism including a set of optical data; calculating a growth index based on the set of optical data; and calculating an anticipated harvest time based on the growth index, where the image information includes at least one of: (a) visible image data obtained from an image sensor and non-visible image data obtained from the image sensor, and (b) a set of image data from at least two image capture devices, where the at least two image capture devices capture the set of image data from at least two positions.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit under 35U.S.C. § 120 of U.S. Pat. Application Serial No. 17/713,639, titled“METHOD, SYSTEM, AND MEDIUM HAVING STORED THEREON INSTRUCTIONS THATCAUSE A PROCESSOR TO EXECUTE A METHOD FOR OBTAINING IMAGE INFORMATION OFAN ORGANISM COMPRISING A SET OF OPTICAL DATA,” filed Apr. 5, 2022, whichclaims the benefit under 35 U.S.C. § 120 of U.S. Pat. Application SerialNo. 16/812,678, titled “METHOD, SYSTEM, AND MEDIUM HAVING STORED THEREONINSTRUCTIONS THAT CAUSE A PROCESSOR TO EXECUTE A METHOD FOR OBTAININGIMAGE INFORMATION OF AN ORGANISM COMPRISING A SET OF OPTICAL DATA,”filed Mar. 9, 2020, which claims the benefit under 35 U.S.C. § 120 ofU.S. Pat. Application Serial No. 14/777,549, titled “METHOD, SYSTEM, ANDMEDIUM HAVING STORED THEREON INSTRUCTIONS THAT CAUSE A PROCESSOR TOEXECUTE A METHOD FOR OBTAINING IMAGE INFORMATION OF AN ORGANISMCOMPRISING A SET OF OPTICAL DATA,” filed Sep. 16, 2015, now U.S. Pat.No. 10,607,078, which is a U.S. National Stage Entry of InternationalApplication No. PCT/JP2014/001497, filed in the Japanese Patent Officeas a Receiving office on Mar. 17, 2014, which claims priority toJapanese Priority Patent Application JP 2013-062017 filed Mar. 25, 2013,each of which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

The presently disclosed technology relates to an information processingsystem, an information processing method of the information processingsystem, an imaging device and an imaging method, and a program, andparticularly to an information processing system, an informationprocessing method of the information processing system, an imagingdevice and an imaging method, and a program that enables calculation ofa proper growth index of agricultural produce and an anticipated properharvest time.

BACKGROUND ART

In satellite remote sensing in which a growth state and a harvest seasonof agricultural produce are estimated by sensing reflected light(near-infrared light) from plants using a sensor mounted in a spacesatellite, it is difficult to acquire data under a night sky or clouds,and it takes several days until data from the satellite comes to hand,and thus, real-time information is hard to get. In addition, since asatellite makes a loop trip and thus getting information of a same spotdepends on the cycle of the satellite, rough information of a wide rangeis obtained, while accurate information of a narrow region is difficultto obtain.

In addition, in near remote sensing that uses a sensor installed on theground, the distance from a target to the sensor is short, andtherefore, there are advantages in that sensing is less affected by theatmosphere than in the satellite remote sensing, data from the targetalone can be acquired by the sensor without interference between thesensor and the target, data can be acquired at preferable times, and thelike. Such a remote sensing technology in which image information isacquired in proximity to a plant, the image information is transmittedto a computer, a vegetation index is computed by the computer, and aproper harvest time is evaluated or anticipated based on the correlationbetween the index and evaluation items such as an amount of fiber hasbeen disclosed (refer to PTL 1).

CITATION LIST Patent Literature

[PTL 1] International Publication No. WO2009/116613

SUMMARY Technical Problem

However, in the technology disclosed in PTL 1 described above, since asingle camera that photographs agricultural produce is provided, whengrowth situations of the produce vary within a farm region, the growthsituations of the entire farm are recognized with occasionallyphotographed growth situations of the produce, and thereby there arecases in which accuracy of evaluation or anticipation of a properharvest time is lowered. Also, prior art technology is limited becauseit fails to address growth of organisms. In addition, it is not possibleto grasp growth situations of many farms located in different regions.

In addition, in the technology disclosed in PTL 1, accuracy inevaluating a growth situation of agricultural produce by computing thevegetation index through an arithmetic operation based on data ofnear-infrared light and red light from image data of the agriculturalproduce using a near-infrared light sensor and a red-light sensor is notsufficiently reliable. In other words, it is difficult to improveaccuracy of evaluation by performing both evaluation in combination withevaluation of a growth situation using colors of the produce andevaluation of a growth situation based on the vegetation index.

Furthermore, the technology disclosed in PTL 1 discloses that adedicated device for remote sensing may be used as a camera. For such adedicated device for remote sensing, a multi-spectral camera (multi-bandcamera) or a hyper-spectral camera is used. The former necessitatesmechanical switching of a band-pass filter, and synchronism of imageregions is insufficient. In addition, since the latter necessitatesscanning, synchronism of image regions is insufficient, further, sincean optical system thereof is complicated, it is difficult to miniaturizethe camera which is expensive, and further, since data take a largecapacity, a communication load increases, and thus the camera is notappropriate for wireless communication.

Furthermore, the technology disclosed in PTL 1 is based on the premisethat the evaluation result or the anticipated proper harvest time shouldbe provided to a producer or a manager. In this case, the producer cananticipate and grasp a harvest time, but it is difficult to satisfydemands of retailers, general consumers, consumers such as restaurants,distributors, or other outside parties who want to purchase agriculturalproduce without going through a retailer and/or who want to know aharvest time of the produce.

It is desirable to be able to properly compute a growth index and ananticipated proper harvest time of agricultural produce based on an RGBimage and an NIR image, and be able to distribute information about thegrowth index and the anticipated proper harvest time not only to aproducer and a manager but also to retailers, general consumers, anddistributors, among others.

Solution to Problem

Various embodiments of the present disclosure relate to methodsincluding: obtaining image information of an organism including a set ofoptical data; calculating a growth index based on the set of opticaldata; and calculating an anticipated harvest time based on the growthindex, where the image information includes at least one of: (a) visibleimage data obtained from an image sensor and non-visible image dataobtained from the image sensor, and (b) a set of image data from atleast two image capture devices, where the at least two image capturedevices capture the set of image data from at least two positions.

Further embodiments relate to systems including: an image capturedevice, where at least one of the server and the image capture device isconfigured to: obtain image information of an organism including a setof optical data; calculate a growth index based on the set of opticaldata; and calculate an anticipated harvest time based on the growthindex, where the image information includes at least one of: (a) visibleimage data obtained from an image sensor and non-visible image dataobtained from the image sensor, and (b) a set of image data from atleast two image capture devices, where the at least two image capturedevices capture the set of image data from at least two positions.

Still further embodiments relate to tangible, non-transitorycomputer-readable mediums having stored thereon instructions that causea processor to execute a method, the method including: obtaining imageinformation of an organism including a set of optical data; calculatinga growth index based on the set of optical data; and calculating ananticipated harvest time based on the growth index, where the imageinformation includes at least one of: (a) visible image data obtainedfrom an image sensor and non-visible image data obtained from the imagesensor, and (b) a set of image data from at least two image capturedevices, where the at least two image capture devices capture the set ofimage data from at least two positions.

As used herein in various illustrative embodiments, the terms “produce”and “agricultural produce” include organisms. An organism is any livingsystem. The living system may be biologically contiguous.

A further definition of organism, as used herein, is an assembly ofmolecules functioning as a more or less stable whole that exhibits theproperties of life, which includes any living structure capable ofgrowth. Thus, for example, an organism includes, but is not limited to,an animal, fungus, micro-organism, and plant.

Therefore, the terms “produce” and variations thereof, including but notlimited to “agricultural produce,” as used herein, include but are notlimited to animals such as cows, goats, sheep, pigs, fish, and poultry.

Accordingly, for example, the terms “growth index” and variationsthereof, including but not limited to “growth state information,”“growth situation information,” include but are not limited to growth oforganisms, including produce and animals.

In addition, for example, the terms “harvest” and variations thereof,including but not limited to “harvesting,” “harvest time information,”“anticipated proper harvest time,” “harvest plan,” “harvest planinformation,” “harvest start time,” “harvest time,” and “harvest timelimit,” refer to harvesting of organisms. In various illustrativeembodiments, harvesting includes any gathering of mature organismsincluding produce and/or animals.

Thus, the term “evaluating a growth situation” and variations thereof,as used herein, includes evaluating a growth situation of organisms suchas animals and produce. Such evaluation may use various properties ofthe animals and produce, including a growth index and other propertiesnot listed explicitly herein.

Methods and systems disclosed herein may use optical data. For example,a set of optical data may be used to obtain growth information or agrowth index. The optical data may include captured image data includingvisible and non-visible image data.

As used herein, the term “visible image data” may include image datausing a red-green-blue (also referred to as RGB) color model. Forexample, digital cameras and video cameras often use a particular RGBcolor space.

As used herein, the term “non-visible image data” may includenear-infrared rays (hereinafter, also referred to as NIR).

As used herein, the term “outside parties” and variations thereof,includes general consumers, retailers, restaurants, and food producers.For example, outside parties may include any person or business relatedto the supply chain system.

An image capture device, as used in various illustrative embodimentsdisclosed herein, is a device that captures image data or imageinformation. For example, an image capture device may include, but isnot limited to, optical devices that store and/or transmit still ormoving image data such as a camera or a video camera.

The term “sensor camera” and variations thereof, as used herein, refersto a device that captures images . Sensor cameras may have variousfunctionalities, such as the ability to collect, send, and/or storevarious properties. Such properties may include but are not limited toinformation related to growth, temperature, humidity, and atmosphericpressure.

In addition, sensor cameras may have the functionality to transferinformation or data over a network or to an external device. Forexample, the sensor cameras may supply information, including capturedimage data, to a server.

In the description herein, for the purposes of illustration, methods maybe described in a particular order. It should be appreciated that inalternate embodiments, the methods may be performed in a different orderthan that described. It should also be appreciated that the methodsdescribed herein may be performed by hardware components or may beembodied in sequences of machine-executable instructions, which may beused to cause a machine, such as a general-purpose or special-purposeprocessor (GPU or CPU) or logic circuits programmed with theinstructions to perform the methods (FPGA). These machine-executableinstructions may be stored on one or more machine readable mediums, suchas CD-ROMs or other type of optical disks, floppy diskettes, ROMs, RAMs,EPROMs, EEPROMs, magnetic or optical cards, flash memory, or other typesof machine-readable mediums suitable for storing electronicinstructions. Alternatively, the methods may be performed by acombination of hardware and software.

Specific details are given in the description to provide a thoroughunderstanding of the embodiments. However, it will be understood by oneof ordinary skill in the art that the illustrative embodiments may bepracticed without these specific details.

For example, in some instances, well-known circuits, processes,algorithms, structures, and techniques may be shown or discussed withoutunnecessary detail in order to avoid obscuring the illustrativeembodiments.

Also, it is noted that the embodiments are described as variousprocesses which may be depicted as a flowchart, a flow diagram, a dataflow diagram, a structure diagram, or a block diagram, among others.Although any of these depictions may describe various parts of theoperations as a sequential process or sequential processes, many of theoperations or parts of the operations can be performed in parallel,concurrently, and or redundantly.

In addition, the order of the operations may be re-arranged. A processis terminated when its operations are completed, but could haveadditional steps or repetitive steps not included in the figure. Aprocess may correspond to a method, a function, a procedure, asubroutine, a subprogram, etc. When a process corresponds to a function,its termination corresponds to a return of the function to the callingfunction or the main function.

Furthermore, embodiments may be implemented by hardware, software,firmware, middleware, microcode, and hardware description languages,among others, or any combination thereof. When implemented in software,firmware, middleware or microcode, the program code or code segments toperform the necessary tasks may be stored in a machine readable mediumsuch as a storage medium.

A processor (s) may perform the necessary tasks. A code segment mayrepresent a procedure, a function, a subprogram, a program, a routine, asubroutine, a module, a software package, a class, or any combination ofinstructions, data structures, or program statements. A code segment maybe coupled to another code segment or a hardware circuit by passingand/or receiving information, data, arguments, parameters, or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

While illustrative embodiments of the disclosure have been described indetail herein, it is to be understood that the inventive concepts may beotherwise variously embodied and employed, and that the appended claimsare intended to be construed to include such variations, except aslimited by the prior art.

Advantageous Effects of Invention

According to the embodiments of the presently disclosed technology, agrowth index and an anticipated proper harvest time of agriculturalproduce can be computed. In various embodiments, the computation of thegrowth index and the anticipated proper harvest time can be improvedover prior art computations.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an illustrative diagram showing a configuration example of aninformation processing system according to various embodiments of thepresently disclosed technology.

FIG. 2 is an illustrative diagram showing a configuration example of asensor camera of FIG. 1 according to various embodiments of thepresently disclosed technology.

FIG. 3 is an illustrative diagram showing a configuration example of asensor in the sensor camera of FIG. 2 according to various embodimentsof the presently disclosed technology.

FIG. 4 is an illustrative diagram showing a configuration example of aterminal device of FIG. 1 according to various embodiments of thepresently disclosed technology.

FIG. 5 is an illustrative diagram showing a configuration example of aserver of FIG. 1 according to various embodiments of the presentlydisclosed technology.

FIG. 6 is an illustrative diagram showing a configuration example ofmanagement information according to various embodiments of the presentlydisclosed technology.

FIG. 7 is an illustrative flowchart for describing a growth situationinformation accumulation process performed by the sensor cameraaccording to various embodiments of the presently disclosed technology.

FIG. 8 is an illustrative diagram for describing a transfer method ofgrowth situation information between sensor cameras according to variousembodiments of the presently disclosed technology.

FIG. 9 is an illustrative diagram for describing another transfer methodof growth situation information between sensor cameras according tovarious embodiments of the presently disclosed technology.

FIG. 10 is an illustrative flowchart for describing a sensing processperformed by the sensor camera according to various embodiments of thepresently disclosed technology.

FIG. 11 is an illustrative flowchart for describing the growth situationinformation accumulation process performed by a server according tovarious embodiments of the presently disclosed technology.

FIG. 12 is an illustrative flowchart for describing a harvest planreception process performed by the terminal device according to variousembodiments of the presently disclosed technology.

FIG. 13 is an illustrative diagram for describing an imaging principleof a stereoscopic image according to various embodiments of thepresently disclosed technology.

FIG. 14 is an illustrative flowchart for describing an inquiry responseprocess between the terminal device and the server according to variousembodiments of the presently disclosed technology.

FIG. 15 is an illustrative diagram for describing a first modificationexample of the sensor according to various embodiments of the presentlydisclosed technology.

FIG. 16 is an illustrative diagram for describing a second modificationexample of the sensor according to various embodiments of the presentlydisclosed technology.

FIG. 17 is an illustrative diagram for describing a third modificationexample of the sensor according to various embodiments of the presentlydisclosed technology.

FIG. 18 is an illustrative diagram for describing a fourth modificationexample of the sensor according to various embodiments of the presentlydisclosed technology.

FIG. 19 is an illustrative diagram for describing a configurationexample of a general-purpose personal computer according to variousembodiments of the presently disclosed technology.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Hereinafter, various illustrative embodiments for the present disclosure(hereinafter, referred to as embodiments) will be described. Note thatdescription will be provided in the following order.

-   1. First embodiment (Configuration example of an embodiment of an    information processing system)-   2. First modification example (First modification example of a    sensor structure)-   3. Second modification example (Second modification example of the    sensor structure)-   4. Third modification example (Third modification example of the    sensor structure)-   5. Fourth modification example (Fourth modification example of the    sensor structure)

1. First Embodiment [Configuration example of an information processingsystem]

First, with reference to FIG. 1 , a configuration example of aninformation processing system that is an illustrative configurationexample of various embodiments of the presently disclosed technologywill be described.

The information processing system of FIG. 1 is configured to includesensor cameras 11-1 to 11-N, terminal devices 12-1 to 12-4 each managedby a consumer, a retailer, a distributor, and a farmer, a network 13,and a server 14. In the information processing system of FIG. 1 , imagescaptured by the sensor cameras 11-1 to 11-N are supplied to the server14 via the network 13 represented by the Internet, and thereby theserver 14 computes a growth index of agricultural produce and computesan anticipated proper harvest time based on the growth index. Inaddition, the server 14 responds to inquiries such as an anticipatedproper harvest time from the terminal devices 12-1 to 12-4 each managedby the consumer, the retailer, the distributor, and the farmer, amongother outside parties.

In more detail, the sensor cameras 11-1 to 11-N are disposed so that anentire farmland can be imaged at predetermined intervals of the farmlandfor agricultural produce to be managed (or so that regions that can beproximate to the entire farmland can be imaged by the sensor cameras11-1 to 11-N as a whole), images that include RGB pixels and pixels ofNIR are captured, and the captured image data is transmitted to theserver 14 via the network 13. In addition, the sensor cameras 11 measureinformation of temperature, humidity, and atmospheric pressure, amongothers, as environment information, and supply the information as wellas the captured image data to the server 14 as growth state information.Note that the sensor cameras 11-1 to 11-N are referred to simply as thesensor cameras 11 unless specified otherwise in this and otherconfigurations.

The terminal devices 12-1 to 12-4 are information processing devicesconfigured as, for example, personal computers, among others (alsoincluding mobile terminals such as so-called smartphones) managedrespectively by a consumer, a retailer, a distributor, and a farmer, andmake inquiries of information of a growth index and an anticipatedproper harvest time, among others, via the network 13, and receive anddisplay response information to the inquires with respect to the server14.

The server 14 acquires and accumulates the growth situation informationbased on the image data, and other information supplied from the sensorcameras 11, and computes a growth index and an anticipated properharvest time based on the image data. In addition, the server 14 alsouses growth situation information of the past in addition to the imagedata supplied from the sensor cameras 11 to compute an anticipatedproper harvest time. Further, when the anticipated proper harvest timecomputed based on the growth situation information comes, the server 14notifies the terminal devices 12-1 to 12-4 managed respectively by theconsumer, the retailer, the distributor, and the farmer of theinformation that the anticipated proper harvest time has come via thenetwork 13. Note that the anticipated proper harvest time may be ananticipated date as a proper day to start harvesting, or a day prior bya predetermined number of days from the anticipated date, or for apredetermined number of days from the day prior by a few days from theanticipated date.

[Configuration Example for Realizing a Function of the Sensor Cameras]

Regarding FIG. 2 , an illustrative configuration example for realizing afunction of the sensor cameras 11 will be described.

Each sensor camera 11 is provided with a sensor 31, an RGB imagegeneration unit 32, an NDVI image generation unit 33, a control unit 34,an IP address storage unit 35, a GPS 36, an environment informationmeasurement unit 37, an RTC 38, a growth situation informationgeneration unit 39, a communication unit 40, and a communication pathspecification unit 41. The sensor 31 is configured as, for example animage sensor, and has a pixel array as illustrated in, for example, FIG.3 . In other words, in the pixel array of the sensor 31, any of greenarrays in a Bayer array composed of general RGB (red, green, and blue)as shown in an image P1 is constituted by near-infrared ray pixels .Note that, in following drawings, the horizontally-striped patternindicates green, the vertically-striped pattern indicates blue, theupward-shaded part indicates red, and the downward-shaded part indicatesnear-infrared rays.

The RGB image generation unit 32 generates an RGB image from imagesignals captured by the sensor 31. In other words, the RGB imagegeneration unit 32 extracts signals of green, red, and blue based on theimage signals captured by the sensor 31 with the pixel arrays as shownin the image P1 of FIG. 3 as shown respectively by images P11 to P13,and thereby generates component signal images of green, red, and blue asshown by images P21 to P23 by demosaicing the signals. Further, the RGBimage generation unit 32 generates an RGB image as shown by an image P31by forming the RGB component signal images as shown by the images P21 toP23.

The Normalized Difference Vegetation Index (NDVI) image generation unit33 generates NIR images from signals of an image captured by the sensor31. In other words, the NDVI image generation unit 33 extracts NIRsignals as shown by an image P14 based on the signals of the imagecaptured by the sensor 31 with the pixel arrays as shown by the image P1of FIG. 3 , and thereby generates an NIR component signal image as shownby an image P24. Furthermore, the NDVI image generation unit 33generates an NDVI image based on the NIR component signal image and ared component signal image generated by the RGB image generation unit 32described above. Note that the Normalized Difference Vegetation Index(NDVI) will be described later in detail.

The control unit 34 is constituted by, for example, a microprocessor, amemory, and the like, executes various processes by performing programsstored in the memory, and accordingly controls the entire operations ofthe sensor cameras 11.

The Internet Protocol (IP) address storage unit 35 stores IP addresseswhich are information for individually identifying the sensor cameras11, and may supply the information of the IP addresses to the controlunit 34. The Global Positioning System (GPS) 36 receives radio wavesfrom satellites not shown in the drawings, computes positionalinformation such as the longitude and the latitude of the earth in whichthe sensor cameras 11 are installed, and supplies the information to thecontrol unit 34. The environment information measurement unit 37measures information of temperature, humidity, and atmospheric pressure,among others as information on the environment in which the sensorcameras 11 are installed, and supplies the information to the controlunit 34. The unit includes a Real Time Clock (RTC), and generates timeinformation at all times and supplies the information to the controlunit 34. Note that, here, the example in which IP addresses are used asinformation for individually identifying the sensor cameras 11 isdescribed, however, as information that can individually identify thesensor cameras 11, information other than the IP addresses may be used.

When the sensor 31 captures an image, the growth situation informationgeneration unit 39 generates growth situation information that includesthe IP addresses, the RGB image, the NDVI image, the positionalinformation, and the environment information together with timeinformation of the capturing timing. Note that information other thanthe IP addresses, the RGB image, the NDVI image, the positionalinformation, and the environment information may be included in thegrowth situation information as long as a growth situation can bechecked with the information.

The communication unit 40 is a unit that performs wired or wirelesscommunication via the network 13 such as the Internet, including, forexample, an Ethernet board, among others, and controlled by the controlunit 34 to transmit the growth situation information to the server 14.The communication path specification unit 41 specifies a communicationpath during transmission of the growth situation information by thecommunication unit 40. In other words, the communication pathspecification unit 41 transmits the growth situation information, whichis to be supplied to the server 14 by numerous sensor cameras 11, to theserver 14 in the form of sequential relaying between the sensor cameras11. In other words, when growth situation information of each of thesensor cameras 11-1 to 11-3 is transmitted, the sensor camera 11-1transmits its growth situation information to the sensor camera 11-2,and the sensor camera 11-2 supplies the growth situation informationsupplied from the sensor camera 11-1 and its own growth situationinformation to the sensor camera 11-3. Furthermore, the sensor camera11-3 supplies the growth situation information of the sensor cameras11-1 and 11-2 and its own growth situation information to the server 14.In order to perform the communication, the communication pathspecification unit 41 specifies a communication path by deciding whichsensor cameras 11 should be passed through to transmit a sensor camera’sgrowth situation information. As one illustrative specific example, whena communication path specification unit 41 of a sensor camera 11communicates with a communication path specification unit 41 of a nearsensor camera 11 through a communication unit 40 and captures imagestogether that make a pair in order to constitute, for example, astereoscopic image to be described later, either sensor camera sets andspecifies a path so as to transmit growth situation information. Withthis process, complexity in terms of the communication path can bereduced, and a communication speed can improve. This form ofcommunication may be the same as near field communication representedby, for example, Zigbee (registered trademark). Note that, acommunication path may be useful if growth situation information piecescan be sequentially transmitted on the path to the server 14 withimproved efficiency, the form of relaying described above being a mereexample, and the information pieces may be transmitted in another form.

[Configuration Example to Realize a Function of Terminal Devices]

Regarding FIG. 4 , an illustrative configuration example to realize afunction of the terminal devices 12 each managed by the consumer, theretailer, the distributor, and the farmer will be described.

Each of the terminal devices 12 managed by the consumer, the retailer,the distributor, and the farmer is configured to include a control unit61, an inquiring unit 62, an operation unit 63, a communication unit 64,an IP address storage unit 65, and a display unit 66. The control unit61 may include a microprocessor and a memory, among other components,and controls entire operations of the terminal device 12 with themicroprocessor executing data and programs stored in the memory. Whenthere is an instruction to make an inquiry of all or some of imagescaptured by the sensor cameras 11, a growth index, and an anticipatedproper harvest time through an operation of the operation unit 63 thatincludes a keyboard and a mouse, among others, the inquiring unit 62controls the communication unit 64 that includes an Ethernet board, forexample, such that inquiry information is generated for making aninquiry of images captured by the sensor cameras 11, a growth index, andan anticipated proper harvest time to the server 14 together withinformation of IP addresses for specifying the sensor cameras 11 whichis stored in the IP address storage unit 65 and managed by the inquiringunit itself (or of which the inquiring unit desires to make an inquiry).The inquiring unit 62 transmits the generated inquiry information to theserver 14 using the communication unit 64. In addition, thecommunication unit 64 receives response information transmitted from theserver 14 in response to the inquiry information and supplies theinformation to the control unit 61. The control unit 61 causes thedisplay unit 66 that includes a Liquid Crystal Display (LCD), and anorganic EL (Electro Luminescence), among others, to display the responseinformation.

[Configuration Example to Realize a Function of the Server]

Regarding FIG. 5 , an illustrative configuration example to realize afunction of the server 14 will be described.

The server 14 is configured to include a control unit 81, a growthsituation information accumulation unit 82, a target regionspecification unit 83, an RGB image growth index computation unit 84, anNDVI image growth index computation unit 85, a stereoscopic image growthindex computation unit 86, a harvest time computation unit 87, amanagement information accumulation unit 88, a growth index computationunit 89, a mapping unit 90, a sensor camera operation situationmonitoring unit 91, a communication unit 92, a harvest plan creationunit 93, a delivery plan creation unit 94, a sales plan creation unit95, a purchase plan creation unit 96, an inquiry reception unit 97, anda response creation unit 98.

The control unit 81 may include a microprocessor, a memory, and thelike, and controls entire operations of the server 14 by executing dataand programs stored in the memory.

The growth situation information accumulation unit 82 stores growthsituation information supplied from the sensor cameras 11 via thecommunication unit 92 in association with IP addresses used to identifythe sensor cameras 11.

The target region specification unit 83 specifies a region within animage in which agricultural produce to be monitored is present based onan RGB image included in growth situation information. As oneillustrative specific example, the target region specification unit 83stores patterns of colors and shapes serving as characteristicinformation of each agricultural produce, and specifies a target regionby searching for a region that matches the characteristic informationwithin the RGB image. Note that, here, the target region specificationunit 83 being described provided in the server 14, may, however, beprovided in each sensor camera 11 so that, for example, information ofthe target region is included in growth situation information. Inaddition, since the target region specification unit 83 has only to beable to specify a target region, the target region specification unitmay specify a target region only using an image other than an RGB image,for example, an NIR image.

The RGB image growth index computation unit 84 computes a growth indexbased on information of an image region specified as a target region outof an RGB image. For example, since the time in which a ratio of greenaccounting for a rice hull in one ear is about 10% is set to be aharvest start time, and the time in which a ratio thereof is about 2% isset to be a harvest time limit, the RGB image growth index computationunit 84 computes a growth index based on the ratios of green of ricehulls. Since the RGB image growth index computation unit 84 computes anRGB image growth index only using image information of a region in anRGB image in which a target is present, the RGB image growth indexcomputation unit 84 can compute the growth index with higher accuracy.

The NDVI image growth index computation unit 85 computes a growth indexbased on information of an image region specified as a target region inan NDVI image. Here, an NDVI indicates a normalized vegetation index asexpressed by the following formula (1).

NDVI = (R_NIR − R_RED)/(R_NIR+R_RED)

In formula (1), NDVI is a normalized vegetation index, R_NIR is thereflectance of near-infrared light, and R_RED is the reflectance of redlight. Thus, the NDVI image generation unit 33 of the sensor camera 11described above generates an image obtained from an arithmetic operationof the above-described formula (1) as an NDVI image. An NDVI is used asan index of growth of foliage. Note that the reflectances ofnear-infrared light and red light are computed by obtaining red lightintensity and NIR intensity in a region that is not a target region, forexample, the sky as incident light intensity, and obtaining red lightintensity and NIR intensity in a target region as reflected lightintensity in an RGB image and an NIR image. In addition, thereflectances of near-infrared light and red light may also be obtainedby measuring intensity of incident light with reference to a diffuserpanel having a known reflectance, calculating a reflection coefficientfrom a ratio between the intensity and reflection luminance of a target,and converting the coefficient to a reflectance. Furthermore, the NDVIimage growth index computation unit 85 computes an NDVI image growthindex from the average value, the variance, or the high-order varianceof an NDVI only of a target region. With the operation, the NDVI imagegrowth index is computed only from information obtained from pixelswithin the target region, and the NDVI image growth index can becomputed with higher accuracy.

The stereoscopic image growth index computation unit 86 generates aparallax image based on information of regions of the same targetcaptured by the plurality of sensor cameras 11, acquires sizes of targetagricultural produce as stereoscopic information, and computes astereoscopic image growth index based on image information that includesthe stereoscopic sizes.

The harvest time computation unit 87 computes an anticipated properharvest time based on an RGB growth index, an NDVI growth index, astereoscopic image growth index, and past information of the informationof the aforementioned elements accumulated in the growth situationinformation accumulation unit 82.

The management information accumulation unit 88 stores information of asensor position, a region (a country, a city, etc.), the type ofagricultural produce, the owner of agricultural produce (or a farm), afarm or field Gp, a contract distributor, a contract retailer, a group,a growth index, and an anticipated proper harvest time for each IPaddress for identifying the sensor cameras 11 as shown in FIG. 6 . Inthe field of the sensor position, information acquired by the GPSs 36provided in the sensor cameras 11 is registered. In the field of aregion, a country, a city, etc. set in association with a sensorposition is registered. In the field of the type of agriculturalproduce, information indicating the type of agricultural producecultivated in a cultivation area monitored by the sensor cameras 11 isregistered. In the field of the owner of agricultural produce (or afarm), information of the owner of agricultural produce or a farm forwhich the sensor cameras 11 specified by IP addresses are installed isregistered. In the field of the farm or field Gp, a group, etc. managedby, for example, the same owner is registered. In the field of thecontract distributor, information of a distributor who will transportagricultural produce monitored by the sensor cameras 11 which areidentified by the IP addresses is registered. In the field of thecontract retailer, information of a contract retailer who will sell theagricultural produce monitored by the sensor cameras 11 which areidentified by the IP addresses is registered. In the field of the group,a group name allotted to regions in which harvesting is performed at thesame time is registered. In the field of growth index, a growth index ofthe agricultural produce within the range monitored by the sensorcameras 11 which are identified by the IP addresses is registered. Inthe field of the anticipated proper harvest time, information of ananticipated proper harvest time that is anticipated based on the growthindex and past information thereof is registered.

In FIG. 6 , AAA, BBB, and CCC are registered as IP addresses. Inaddition, a sensor position with an IP address of AAA is indicated by A,the region by a, the type of agricultural produce by Alpha, the owner ofagricultural produce by “Kou,” the farm or field Gp by G1, the contractdistributor by (1), the contract retailer by “Ah,” the group by i, thegrowth index by 60, and the anticipated proper harvest time by October15.

In a similar manner, a sensor position with an IP address of BBB isindicated by B, the region by a, the type of agricultural produce byAlpha, the owner of agricultural produce by “Kou,” the farm or field Gpby G1, the contract distributor by (1), the contract retailer by “Ah, ”the group by i, the growth index by 70, and the anticipated properharvest time by October 16.

Furthermore, a sensor position with an IP address of CCC is indicated byC, the region by c, the type of agricultural produce by Beta, the ownerof agricultural produce by “Otsu,” the farm or field Gp by G2, thecontract distributor by (2), the contract retailer by “Eah, ” the groupby ii, the growth index by 65, and the anticipated proper harvest timeby October 20.

The growth index computation unit 89 computes a growth index set as, forexample, a weighted average of an RGB growth index, an NDVI growthindex, and a stereoscopic image growth index based on any one or all ofthe indexes.

The mapping unit 90 generates information obtained by mapping growthindexes and anticipated proper harvest times as information on maps ofeach region.

When the sensor camera operation situation monitoring unit 91 comparestime-series changes of RGB images included in growth situationinformation and there are extremely drastic changes, the sensor cameraoperation situation monitoring unit monitors operation situations bydetermining whether or not there is an abnormal operation stateoccurring in the sensor cameras 11.

The communication unit 92 may include an Ethernet board, and the like,and is controlled by the control unit 81, thereby receiving the growthsituation information and the inquiry information transmitted from theterminal devices 12 and transmitting response information to theterminal devices 12.

The harvest plan creation unit 93 generates harvest plan informationfrom harvest time information based on the growth situation informationand anticipated proper harvest time information, and transmits theinformation to the terminal device 12 managed and operated by the farmerby using the communication unit 92. Note that the harvest planinformation may be transmitted not only to the terminal device 12managed and operated by the farmer but also to the terminal devices 12managed and operated by the distributor, the retailer, and the consumer.Owing to the transmission, the distributor, the retailer, and theconsumer also can formulate their own distribution plans, sales plans,and purchase plans from the harvest plan information.

The delivery plan creation unit 94 generates delivery plan informationfrom the harvest time information based on the growth situationinformation and the anticipated proper harvest time information, andtransmits the information to the terminal device 12 managed and operatedby the distributor by using the communication unit 92.

The sales plan creation unit 95 generates sales plan information fromthe harvest time information based on the growth situation informationand the anticipated proper harvest time information, and transmits theinformation to the terminal device 12 managed and operated by theretailer by using the communication unit 92.

The purchase plan creation unit 96 generates purchase plan informationfrom the harvest time information based on the growth situationinformation and the anticipated proper harvest time information, andtransmits the information to the terminal device 12 managed and operatedby the consumer by using the communication unit 92.

The inquiry reception unit 97 controls the communication unit 92 suchthat inquiry information is received including inquiries with regard toa harvest time, among other information, transmitted from the terminaldevices 12 operated by either of the consumer, the retailer, thedistributor, and the farmer, for example, through the network 13.

The response creation unit 98 generates response information including,for example, growth index mapping information generated by the mappingunit 90 corresponding to information received as inquiry information,and controls the communication unit 92 such that the responseinformation is transmitted to the terminal devices 12 that transmittedthe inquiry information.

[Growth Situation Information Accumulation Process by a Sensor Camera]

Regarding FIG. 7 , an illustrative growth situation informationaccumulation process by a sensor camera 11 will be described.

In Step S11, the control unit 34 of the sensor camera 11 determineswhether or not a predetermined time has elapsed from a previous sensingprocess based on time information generated by the RTC 38 and timeinformation at which the previous sensing process started. When apredetermined time has not elapsed from the previous sensing process inStep S11, the process proceeds to Step S15.

In Step S15, the control unit 34 determines whether or not an end of anoperation has been instructed through an operation of an operation unitnot shown in the drawings. When an end of the operation is instructed inStep S15, the process ends, and when an end of the operation is notinstructed, the process returns to Step S11. In other words, theprocesses of Steps S11 and S15 are repeated until an end of theoperation is instructed or the predetermined time elapses. In addition,when the predetermined time elapses in Step S11, the process proceeds toStep S12.

In Step S12, the sensor 31 executes a sensing process, and acquires anRGB image and an NDVI image from the sensing process. Note that variousillustrative embodiments of the sensing process will be described laterin detail with reference to the flowchart of FIG. 10 .

In Step S13, the growth situation information generation unit 39generates growth situation information based on the RGB image and theNDVI image acquired from the sensing process, IP addresses stored in theIP address storage unit 35, positional information including longitudeand latitude on the earth acquired by the GPS 36, information oftemperature, humidity, and atmospheric pressure measured by theenvironment information measurement unit 37 and the time informationgenerated by the RTC 38. Note that, since the growth situationinformation has only to include information indicating a growthsituation of agricultural produce or information for recognizing thegrowth situation, the growth situation information may includeinformation indicating a growth situation or information for recognizingthe growth situation in addition to the RGB image and the NDVI image,the IP address, the positional information including longitude andlatitude on the earth, the information of temperature, humidity, andatmospheric pressure, and the time information.

In Step S14, the control unit 34 controls the communication unit 40 suchthat the generated growth situation information is transmitted to theserver 14, and the process returns to Step S11. At this moment, invarious embodiments, the control unit 34 controls the communication pathspecification unit 41 so as to communicate with a peripheral sensorcamera 11, then to specify a sensor camera 11 which will be passedthrough for transmitting the growth situation information to the server14, and then to transmit the growth situation information to the server14 via the specified sensor camera 11 on the communication path.

In other words, as illustrated in FIG. 8 , when positions in which thesensor cameras 11 are installed are indicated by nodes N1 to N10, forexample, and information is set to be output to the Internet via a basestation K and a gateway GW, growth situation information of the sensorcamera 11 corresponding to the node N5 is transferred to the basestation K through the sensor camera 11 indicated by the node N4 and thesensor camera 11 indicated by the node N3 which are located nearby. Inthis illustrative case, the sensor camera 11 indicated by the node N4transfers the growth situation information of the node N5 and its owngrowth situation information to the node N3. Furthermore, the nodes N1and N2 transfer their own growth situation information to the Node N3.In addition, the node N3 rearranges, and transfers to the base stationK, the growth situation information from the nodes N1 to N5. Inaddition, the sensor camera 11 indicated by the node N7 transfers growthsituation information to the sensor camera 11 indicated by the node N6,and the sensor camera 11 indicated by the node N6 rearranges the growthsituation information of the nodes N6 and N7, and outputs theinformation to the Internet via the base station K and the gateway GW.Furthermore, the sensor cameras 11 indicated by the nodes N9 and N10respectively transfer their growth situation information to the sensorcamera 11 indicated by the node N8, and the sensor camera 11 indicatedby the node N8 rearranges the growth situation information of the nodesN8 to N10, and outputs the information to the Internet via the basestation K and the gateway GW.

Due to the above process, complexity caused by communication between thebase station K and the gateway GW can be more relieved and the growthsituation information can be transferred at a higher speed than when thegrowth situation information from all sensor cameras 11 is output at onetime. Note that since the growth situation information of all sensorcameras 11 has only to be transmitted to the server 14 with efficiency,the growth situation information of all sensor cameras 11 may betransferred using methods other than being transferred between thesensor cameras 11 in the form of relaying as described above, or, forexample, may be transferred directly to the base station K from each ofthe sensor cameras 11. In addition, each of the sensor cameras 11 mayrearrange and transfer the growth situation information from anothersensor camera 11, or may sequentially transfer each piece of growthsituation information to the base station K in a predetermined order. Asone example, when each of the sensor cameras 11 transfers the growthsituation information directly to the base station K, the informationcan be transferred from each of the sensor cameras 11 to the basestation K, and this may happen with improved efficiency.

In addition, as illustrated in FIG. 9 , when sensor cameras 11 indicatedby nodes N11 to N17, N21 to N23, and N31 to N33 are installed, it may beconfigured that, with regard to the sensor cameras 11 indicated by, forexample, the nodes N21 to N23 and N31 to N33, the sensor cameras 11indicated by the nodes N21 to N23 are set to be a first group G11, thesensor cameras 11 indicated by the nodes N31 to N33 are set to be asecond group G12, pieces of the growth situation information arecollected in a representative node of each group, and the sensor camera11 of the representative node rearranges and outputs pieces of thegrowth situation information of the sensor cameras 11 of the other nodesthat belong to the group. Further, with regard to setting the groups G11and G12, for example, sensor cameras 11 present in farmlands owned bythe same owner may be set to be in the same group, or sensor cameras 11paired in order to capture a stereoscopic image described herein may beset to be in the same group.

From the process described above, pieces of the growth situationinformation that include RGB images and NDVI images can be generated ata predetermined time interval, sequentially transmitted to the server14, and sequentially accumulated in the server 14.

[Sensing Process]

Regarding FIG. 10 , an illustrative sensing process will be described.

In Step S31, the sensor 31 captures an image having a size in which thesize and the color of agricultural produce to be harvested can be fullyrecognized in a range of cultivation of the produce that is a subject.In addition, the sensor cameras 11 are installed at an interval and in adirection in which imaging of the farmland under the imaging conditionsdescribed above can be performed.

In Step S32, the RGB image generation unit 32 and the NDVI imagegeneration unit 33 performs a demosaicing process on light beams of eachcolor of pixels captured by the sensor 31. In other words, the RGB imagegeneration unit 32 performs the demosaicing process on pixels ofrespective red, green and blue light to generate red, green, and bluecomponent signal images. In addition, the NDVI image generation unit 33performs a demosaicing process on pixels of NIR to generate an NIRcomponent signal image.

In Step S33, the RGB image generation unit 32 combines the demosaicedRGB component signal images to generate an RGB image.

In Step S34, the NDVI image generation unit 33 measures intensity of NIRand red light serving as light incident from a region recognized as animage of the sky for each pixel and measures intensity of NIR and redlight serving as reflected light in regions other than theaforementioned region based on the NIR image and the red image, computesreflectances of the NIR and the red light, and generates an NDVI image.For this reason, the sensor 31 is installed at an angle at which thecaptured region of the agricultural produce which is a subject and aregion in which incident light of red light or NIR from the sky can bemeasured are included. In addition, when it is difficult to install thesensor at this angle, a panorama and a tilt mechanism are provided inthe sensor cameras 11, incident light of red light and NIR are capturedwith the cameras facing the sky, the cameras are controlled such thatregions of the agricultural produce which is a subject are imaged,reflected light is captured, and the NDVI image described above isgenerated. In addition, the reflectances of the NIR and the red lightmay also be obtained by measuring intensity of the incident light withreference to a diffuser panel having a known reflectance, calculating areflection coefficient from a ratio between the intensity and reflectionluminance of a target, and converting the coefficient to a reflectance.

In Step S35, the control unit 34 controls the environment informationmeasurement unit 37 such that temperature, humidity, and atmosphericpressure constituting the environment information are measured.

With the process described above, information constituting growthsituation information such as the RGB image, the NDVI image, andtemperature, humidity, and atmospheric pressure which are included inthe measured environment information is generated. Note that theinformation constituting the growth situation information may includeinformation other than the RGB image, the NDVI image, and temperature,humidity, and atmospheric pressure which are included in the environmentinformation. Such information may include information that is necessaryfor recognizing a growth situation.

[Growth Situation Information Accumulation Processes by The Server andEach Terminal Device]

Regarding FIGS. 11 and 12 , growth situation information accumulationprocesses by the server 14 and each terminal device 12 will bedescribed.

In Step S61, the control unit 81 of the server 14 controls thecommunication unit 92 such that whether or not growth situationinformation has been transmitted from any sensor camera 11 isdetermined, and when the information is not transmitted, the processproceeds to Step S82.

In Step S82, the control unit 81 determines whether or not an operationunit not shown in the drawings has been operated and an end of anoperation has been instructed, and when an end thereof is instructed,the process ends. In addition, when an end thereof is not instructed,the process returns to Step S61. In other words, when an end thereof isnot instructed and the growth situation information is not transmitted,the processes of Steps S61 and S82 are repeated. When the growthsituation information is transmitted in Step S61 from, for example, theprocess of Step S14 of FIG. 7 , the process proceeds to Step S62.

In Step S62, the control unit 81 controls the communication unit 92 suchthat the growth situation information transmitted from the sensorcameras 11 is received and controls the growth situation informationaccumulation unit 82 such that the received growth situation informationis accumulated. In this illustrative case, the received growth situationinformation may be constituted by a plurality of pieces of growthsituation information from the plurality of sensor cameras 11 asdescribed herein. Thus, the plurality of pieces of growth situationinformation may be accumulated through a one-time process. However, inthe following description, the whole process proceeds by assuming thatpieces of growth situation information are transmitted from two sensorcameras 11, which capture the same target as a stereoscopic image, inone reception process, although other processes are also encompassed byvarious embodiments of the present disclosure.

In Step S63, the control unit 81 controls the target regionspecification unit 83 such that a target region that is a region in animage which is obtained by imaging target agricultural produce based onan RGB image included in the transmitted growth situation information isspecified. As one illustrative specific example, the target regionspecification unit 83 extracts feature information such as the shape andthe hue of the agricultural produce of which a growth index is to becomputed from the RGB image. In addition, the target regionspecification unit 83 determines whether or not the extracted featureinformation matches the actual shape and hue of the agricultural producestored in advance, and specifies a target region that includes theregion within the matching RGB image in which the agricultural produceof which a growth index is to be computed is imaged. Note that, in thisillustrative case, with regard to the agricultural produce to bespecified, for example, the target region specification unit 83 mayspecify the feature information by searching for the managementinformation accumulated in the management information accumulation unit88 based on the IP addresses included in the growth situationinformation including the RGB image, and reading and using theinformation registered in the field of the type of agricultural produce,as shown in FIG. 6 .

In Step S64, the control unit 81 controls the RGB image growth indexcomputation unit 84 such that an RGB image growth index is computedbased on the target region in the RGB image. As one illustrativespecific example, with regard to a harvest time of rice, for example,the RGB image growth index computation unit 84 assumes the time in whicha ratio of green accounting for a rice hull in one ear is about 10% tobe a harvest start time, and the time in which a ratio thereof is about2% to be a harvest time limit, and thus, a growth index is computedbased on the ratios of green of rice hulls, and the index is defined asan RGB image growth index.

In Step S65, the control unit 81 controls the NDVI image growth indexcomputation unit 85 such that an NDVI image growth index is computedbased on the target region in an NDVI image. As one illustrativespecific example, the NDVI image growth index computation unit 85computes, for example, the average value, the variance, or thehigh-order variance of an NDVI of the target region, thereby computingthe NDVI image growth index.

In Step S66, the control unit 81 controls the stereoscopic image growthindex computation unit 86 such that a stereoscopic image growth index iscomputed based on a stereoscopic image. As one illustrative specificexample, the stereoscopic image growth index computation unit 86extracts two RGB images included in the growth situation informationfrom at least two sensor cameras 11 which capture RGB images toconstitute the stereoscopic image. In other words, as illustrated inFIG. 13 , the sensor cameras 11-1 and 11-2 image the same agriculturalproduce M1 from different angles, and the stereoscopic image growthindex computation unit 86 generates a stereoscopic image, i.e., aparallax image from two RGB images captured by the two sensor cameras11-1 and 11-2. Furthermore, the stereoscopic image growth indexcomputation unit 86 generates a three-dimensional image of agriculturalproduce present in the target region based on the parallax image, andcomputes a stereoscopic image growth index from the size. Note that thetarget region in the processes of Steps S65 and S66 may be a regionother than a region obtained based on the RGB images as long as a regionof produce that is a target can be specified therefrom, or a region,which, for example, belongs to any of a region of the NDVI image havinga high probability of a target present therein and having an NDVI valuehigher than a predetermined value and a region obtained based on the RGBimages, may be set to be the target region.

In Step S67, the control unit 81 controls the growth index computationunit 89 such that a growth index of the target agricultural produce iscomputed based on the RGB image growth index, the NDVI image growthindex, and the stereoscopic image growth index. As one illustrativespecific example, the growth index computation unit 89 may assume theaverage of the three kinds of growth indexes as a growth index, mayassume the weighted sum of the indexes as a growth index, or may selectone of the indexes as a growth index. In addition, when it is notpossible to compute all of the RGB image growth index, the NDVI imagegrowth index, and the stereoscopic image growth index, the average valueor weighted sum of computable growth indexes may be set as a growthindex.

In Step S68, the control unit 81 searches for management informationamong the management information accumulated in the managementinformation accumulation unit 88, which corresponds to the IP addressincluded in the growth situation information transmitted from the sensorcameras 11, and updates the growth index included in the searchedmanagement information to a value computed from the above-describedprocess.

In Step S69, the control unit 81 controls the harvest time computationunit 87 such that a harvest time is computed based on the growth index,environment information, and growth situation information andinformation of harvest times of the past. In other words, the harvesttime computation unit 87 computes a harvest time of this seasonanticipated from information of a change in a growth evaluation index ofthis season based on the relationship between information of a change ingrowth evaluation indexes and information of harvest times of the pastas an anticipated proper harvest time.

In Step S70, the control unit 81 searches for management informationamong the management information accumulated in the managementinformation accumulation unit 88, which corresponds to the IP addressincluded in the growth situation information transmitted from the sensorcameras 11, and updates information of the anticipated proper harvesttime included in the searched management information to a value computedfrom the above-described process.

In Step S71, the control unit 81 controls the sensor camera operationsituation monitoring unit 91 such that whether or not there is anabnormality occurring in the sensor cameras 11 that transmit the growthsituation information based on the RGB images is determined. As oneillustrative specific example, the sensor camera operation situationmonitoring unit 91 compares a current RGB image to an RGB image capturedat the previous timing both having the same IP address included in thetransmitted growth situation information among the growth situationinformation accumulated in the growth situation information accumulationunit 82, and determines whether or not there is an abnormality occurringin the sensor cameras 11 based on whether or not a change between theimages is greater than a predetermined value. In other words, the sensorcameras 11 are basically fixed-point cameras, and there will not be asignificant change in the RGB images even if a predetermined time of animaging interval is, for example, about one day. Thus, if there is asignificant change, it is considered that a problem has arisen in thesensor cameras 11. Therefore, when the sensor camera operation situationmonitoring unit 91 compares the currently transmitted RGB image to theprevious RGB image, and regards that there is an abnormality occurringin the cameras due to a significant change between the images, theprocess proceeds to Step S72. Note that the occurrence of an abnormalityin the sensor cameras 11 can also be determined from a comparison of NIRimages, NDVI images, NDVI average values, variances, high-ordervariances, and growth indexes.

In Step S72, the sensor camera operation situation monitoring unit 91regards that there is an abnormality occurring in the sensor cameras 11that transmit the growth situation information, searches for managementinformation based on the IP address of the sensor cameras 11, andnotifies a terminal device 12 managed and operated by the owner of theagricultural produce (farmland) included in the searched managementinformation or a mobile telephone not shown in the drawings of theoccurrence.

On the other hand, when it is regarded that no abnormality occurs inStep S71, the process of Step S72 is skipped.

In Step S73, the control unit 81 determines whether or not the growthindex is higher than a predetermined threshold value, and theanticipated proper harvest time is coming. In Step S73, when, forexample, the growth index is higher than the predetermined thresholdvalue and the anticipated proper harvest time is coming, in other words,when it is regarded that the very day corresponding to the anticipatedproper harvest time or a day a predetermined number of days earlier thanthe very day is coming, the process proceeds to Step S74.

In Step S74, the control unit 81 controls the harvest plan creation unit93 such that a harvest plan is created. For example, the harvest plancreation unit 93 estimates an amount of crops to harvest from a range inwhich anticipated proper harvest times overlap in management informationmanaged according to IP addresses, and makes a harvest schedule of howto proceed with the harvesting process from a harvest start day based onprocessing performance of agricultural machines for harvesting which maybe registered in advance by the owner of the same agricultural produce(farmland).

In Step S75, the control unit 81 controls the communication unit 92 suchthat the information of the harvest plan created by the harvest plancreation unit 93 is transmitted to the terminal device 12 managed andoperated by the farmer. Note that the information of the harvest planmay also be transmitted to the terminal devices 12 managed and operatedby the distributor, the retailer, and the consumer. With this operation,the distributor, the retailer, and the consumer can make their owndistribution plan, sales plan, and purchase plan from the information ofthe harvest plan.

After this process, in Step S91 (of FIG. 12 ), the control unit 61 ofeach terminal device 12 controls the communication unit 64 such thatwhether or not the harvest plan has been transmitted is determined, andthe same process is repeated until the plan is transmitted. When theharvest plan is transmitted through, for example, the process of StepS75 of FIG. 12 , in Step S91, the process proceeds to Step S92.

In Step S92, the control unit 61 controls the communication unit 64 suchthat the transmitted information of the harvest plan is received.

In Step S93, the control unit 61 causes the information of the harvestplan received by the communication unit 64 to be displayed on thedisplay unit 66.

In Step S76 (of FIG. 11 ), the control unit 81 controls the deliveryplan creation unit 94 such that a delivery plan is created. For example,the delivery plan creation unit 94 estimates an amount of crops toharvest from a range in which anticipated proper harvest times overlapin management information managed according to IP addresses, and makes adelivery schedule of how to proceed with the delivery process from theharvest start day based on transportation performance of deliveryvehicles for delivery which may be registered in advance by the contractdistributor.

In Step S77, the control unit 81 controls the communication unit 92 suchthat the information of the delivery plan generated by the delivery plancreation unit 94 is transmitted to the terminal device 12 managed andoperated by the contract distributor. Note that since a processperformed in the terminal device 12 is merely reception and display ofthe delivery plan instead of the harvest plan in the process describedwith reference to the flowchart of FIG. 12 , description of the processis omitted. In addition, in the processes of Steps S76 and S77, adelivery plan, a sales plan, and a purchase plan of all farmlandsincluding farmlands on which contracts are not made with thedistributor, the retailer, and the consumer may be transmitted to thedistributor, the retailer, and the consumer who are shown in FIG. 1 ascontractors of a series of services. Furthermore, in the processes ofSteps S76 and S77, a delivery plan, a sales plan, and a purchase planfor farmlands within regions that are within business scopes of thedistributor, the retailer, and the consumer may be transmitted to thedistributor, the retailer, and the consumer who are shown in FIG. 1 ascontractors of a series of services. In such an illustrative case, for abig distribution company, for example, the delivery plan may betransmitted to branches thereof.

In Step S78, the control unit 81 controls the sales plan creation unit95 such that a sales plan is created. As one illustrative specificexample, the sales plan creation unit 95 estimates an amount of crops toharvest from a range in which anticipated proper harvest times overlapin management information managed according to IP addresses, and makes asales schedule of how to proceed sales from the harvest start day basedon an amount of produce displayable in a store front which is registeredin advance by the contract retailer.

In Step S79, the control unit 81 controls the communication unit 92 suchthat information of the sales plan created by the sales plan creationunit 95 is transmitted to the terminal device 12 managed and operated bythe contract retailer. Note that since a process performed in theterminal device 12 is merely reception and display of the sales planinstead of the harvest plan in the process described with reference tothe flowchart of FIG. 12 , description thereof is omitted. In addition,in the processes of Steps S78 and S79, it may be configured that aretailer located near a farmland is selected from retailers shown inFIG. 1 serving as contractors of a series of services for each farmland,and the sales plan is transmitted to the selected retailer. In such anillustrative case, for example, the sales plan may be transmitted tobranches of a big retailer such as a supermarket, and further,information to which a delivery plan is added may be transmittedthereto.

In Step S80, the control unit 81 controls the purchase plan creationunit 96 such that a purchase plan is created. As one illustrativespecific example, the purchase plan creation unit 96 estimates an amountof crops to harvest from a range in which anticipated proper harvesttimes overlap in management information managed according to IPaddresses, and makes a purchase schedule of how to proceed purchase fromthe harvest start day based on a desired amount of produce to purchasewhich is registered in advance by the contract consumer.

In Step S81, the control unit 81 controls the communication unit 92 suchthat information of the purchase plan generated by the purchase plancreation unit 96 is transmitted to the terminal device 12, and theprocess returns to Step S61. Note that since a process performed in theterminal device 12 is merely reception and display of the purchase planinstead of the harvest plan in the process described with reference tothe flowchart of FIG. 12 , description thereof is omitted. In addition,in the processes of Steps S80 and S81, the purchase plan for a farmlandof produce to be purchased may be transmitted to each consumer out ofconsumers shown in FIG. 1 that are contractors of a series of services.In addition, the purchase plan may be generated in accordance with asales plan of a specific retailer, for example, who is located near thelocation of a consumer and transmitted the plan to the consumers whopurchase the produce from the retailer.

In addition, in Step S73, when the growth index is not higher than thepredetermined threshold value, the process proceeds to Steps S83 to S86.Note that since the processes of Steps S83 to S86 are the same as theprocesses of Steps S74 S76, S78, and S80, description thereof isomitted. In other words, even when the growth index is not higher thanthe predetermined threshold value in Step S73, it may be configured thatthe harvest plan, the delivery plan, the sales plan, and the purchaseplan are created, and when there is an inquiry with regard to each ofthe plans, a response may be made to each of the plans, and developmentsof the growth index may be transmitted whenever the developments takeplace.

In other words, from the above process, pieces of the growth situationinformation that are transmitted from the sensor cameras 11 at apredetermined time interval are sequentially accumulated. During thattime, the growth index and the anticipated proper harvest time in themanagement information managed based on the IP addresses of the sensorcameras 11 are sequentially updated and recorded based on pieces of thegrowth situation information. As a result, the growth index, and theanticipated proper harvest time are updated to the latest information ata predetermined time interval, and when the anticipated proper harvesttime is coming, the updated information can be transferred to theterminal devices 12 managed and operated by each of the farmer, thecontract distributor, the contract retailer, and the contract consumerin real-time as information of the harvest plan, the delivery plan, thesales plan, and the purchase plan. In addition, since proper behaviorscan be taken in accordance with the anticipated proper harvest time bytransferring the information of the harvest plan, the delivery plan, thesales plan, and the purchase plan, efficient harvest, delivery, sales,and purchase are possible. Furthermore, by comparing the RGB images in atime series manner thereby detecting an abnormality in the sensorcameras 11, whether or not the sensor cameras 11 are appropriatelyinstalled can be monitored. In addition, since the sensor cameras 11 canbe used as a communication path even when a sensor camera 11 is stolenor moved to another position due to a storm, or when crops are stolen,an abnormality can be detected by comparing transmitted positionalinformation to the positional information included in the managementinformation. In addition, when the sensor cameras 11 can be used as acommunication path even when an imaging direction or angle of the sensorcameras 11 is changed due to any cause, an abnormality can be assumed tooccur and detected through the same process. Note that the harvest plan,the delivery plan, the sales plan, and the purchase plan may betransferred on the very day specified as the anticipated proper harvesttime, or a day a predetermined number of days earlier from the specifiedday.

[Inquiry Response Process]

Regarding FIG. 14 , an inquiry response process will be described. Notethat, here, processes of transmitting inquiry information for making aninquiry of a harvest time to the server 14 by the terminal device 12managed and operated by the farmer, and receiving and displaying theresponse information will be described. Note that, since the sameprocess is performed when the contract retailer, the contract consumer,and the contract distributor make the same inquiry, description thereofis omitted.

In other words, in Step S101, the control unit 61 of the terminal device12 controls the operation unit 63 such that whether or not an inquiryoperation is made according to an operation by a user is determined, andthe same process is repeated until it is regarded that there is aninquiry operation. When it is regarded that there is an inquiryoperation in Step S101, the process proceeds to Step S102.

In Step S102, the inquiring unit 62 generates the inquiry informationfor making an inquiry of a harvest time associated with the IP addressfor identifying the sensor cameras 11 monitoring crops cultivated in thefarmland of the contract farmer stored in the IP address storage unit65. Note that, when the contract distributor, retailer, and consumershown in FIG. 1 who receive a series of services conclude a contract fora service, not for each farmland (each IP address), the inquiring unit62 stores a relationship table in advance in which business scopes ofthe distributor, the retailer, and the consumer are associated with theIP addresses corresponding to the sensor cameras 11, and generatesinquiry information including the IP addresses according to the businessscopes of the distributor, the retailer, and the consumer.

In Step S103, the control unit 61 controls the communication unit 64such that the inquiry information generated by the inquiring unit 62 tomake an inquiry of the harvest time is transmitted to the server 14 viathe network 13.

In Step S121, the control unit 81 of the server 14 controls the inquiryreception unit 97 such that whether or not the inquiry information hasbeen transmitted from the communication unit 92 is determined, and thesame process is repeated until the information is transmitted. Inaddition, when it is regarded that the inquiry information istransmitted in Step S121, the process proceeds to Step S122.

In Step S122, the control unit 81 controls the inquiry reception unit 97such that the inquiry information transmitted from the communicationunit 92 is acquired, and the content of the inquiry is checked.

In Step S123, the control unit 81 searches for the managementinformation accumulated in the management information accumulation unit88 based on the IP address included in the inquiry information, andreads information of an anticipated proper harvest time and informationof regions in the searched management information. Here, when there area plurality of sensor cameras 11 monitoring crops cultivated by thefarmer, a plurality of IP addresses are included. In addition, here, thesearched information of the anticipated proper harvest time is based notonly on a sensor camera 11 installed in the farmer’s own farmland, butalso on an IP address specifying a sensor camera 11 designated by auser.

In Step S124, the control unit 81 controls the mapping unit 90 such thatthe information of the anticipated proper harvest time is mapped inpositions corresponding to the read information of the regions accordingto a schedule, and thereby anticipated proper harvest time mappinginformation is generated.

In Step S125, the control unit 81 controls the response creation unit 98such that response information that includes the generated anticipatedproper harvest time mapping information is created.

In Step S126, the control unit 81 controls the communication unit 92such that the response information that includes the created anticipatedproper harvest time mapping information is transmitted to the terminaldevice 12 that transmitted the inquiry information.

In Step S104, the control unit 61 of the terminal device 12 controls thecommunication unit 64 such that the response information is received.

In Step S105, the control unit 61 of the terminal device 12 causes theresponse information that includes the received anticipated properharvest time mapping information to be displayed on the display unit 66.

From the processes described above, the information of the anticipatedproper harvest time can acquire the mapped information. In addition,with the displayed mapping information, the information of theanticipated proper harvest time can be transferred in real-timeon-demand. Note that, in the above, the inquiry response processperformed by the terminal device 12 managed by the farmer has beendescribed, but the same process can also be performed in the terminaldevices 12 managed by the contract distributor, the contract retailer,and the contract consumer. In addition, in the above, the content of theinquiry is the information of the anticipated proper harvest time, buteven if the content is other information, for example, the harvest plan,the delivery plan, the sales plan, and the purchase plan managed by theserver 14, an inquiry may be made through the same process, or aresponse may be made to the inquiry. Furthermore, it is possible that,for example, mapping information of anticipated proper harvest times ofcrops is transmitted to the terminal devices 12 managed by the farmer,the distributor, and the consumer, or retailers are classified for eachagricultural produce and mapping information of anticipated properharvest times is transmitted to the retailers. In other words, theinformation of the anticipated proper harvest times, for example, thename of agricultural produce, the time to harvest, and the name of abranch to be delivered to can be transmitted to a big supermarket.

Note that, in the above, the example in which RGB images and an NIRimage are captured by the sensor cameras 11, growth situationinformation including the images is transmitted to the server 14, an RGBimage growth index and an NDVI image growth index that are computed bythe server 14, and an anticipated proper harvest time that is computedhave been described. However, by providing the same function of theserver 14 in the sensor cameras 11, it may be configured that, by thesensor cameras 11, a region in which target agricultural produce isimaged is specified, an RGB image growth index and an NDVI image growthindex of a region specified as the region of the RGB images and the NDVIimage in which the agricultural produce is imaged are computed, growthsituation information including the indexes are generated, and theinformation is supplied to the server 14. In addition, in addition tothese functions, the sensor cameras 11 may capture a stereoscopic imagein cooperation with another sensor camera 11 nearby, and compute astereoscopic image growth index. Furthermore, the sensor cameras 11 maycompute a growth index and an anticipated proper harvest time based onthe RGB image growth index, the NDVI image growth index, and thestereoscopic image growth index obtained as above. In this illustrativecase, when the anticipated proper harvest time is computed, the sensorcameras 11 may read growth situation information of the past accumulatedin the server 14, and compute the anticipated proper harvest time alsousing the growth situation information of the past.

In addition, in the above, the configuration example in which the sensorcameras 11, the terminal devices 12, and the server 14 are included hasbeen described for the configuration of the information processingsystem, however, a cloud computer may be used instead of the server 14.

2. First Modification Example

In the above, the example in which information captured by the sensorcameras 11 is demosaiced to generate RGB and NIR component signal imageshas been described, however, as illustrated in FIG. 15 , for example,using images P112 and P114 that include red light signals and NIRsignals before being demosaiced, an NDVI image P132 as shown by an imageP132 may be generated. Since the demosaicing process can be omitted, orthe number of pixels to be dealt with can be reduced with thisoperation, a processing load can be lowered and a processing speed canimprove. Note that, since the images P111, P113, P121 to P123, and P131are the same as the images P11, P13, P21 P31 of FIG. 3 , descriptionthereof will be omitted.

3. Second Modification Example

In addition, in the above, the example in which pixels of the RGB andNIR component signals are arrayed in a direction of a plane of thesensor cameras 11 has been described, however, as illustrated in FIG. 16, for example, the sensor 31 may be configured by laminating sensorlayers perpendicular to a light traveling direction so as to generatecomponent signal images. In other words, in FIG. 16 , a blue lightsensor layer L1, a green light sensor layer L2, a red light sensor layerL3, and an NIR sensor layer L4 are configured to be laminated from abovethe drawing as shown by an image P151. Each layer has a sensor structurein which only a color component with a target wavelength is detected. Asa result, images P161 to P163 that include green light, red light, bluelight, and NIR component signal images of the images P161 to P164 ofeach layer are generated. As a result, an RGB image P171 is generatedfrom the images P161 to P163, and an NDVI image P172 is generated fromthe images P162 and P164.

4. Third Modification Example

Furthermore, with regard to the sensor 31 that detects RGB componentsignals, it may be configured that, for example, an IR cut filter F thatincludes a dielectric laminated film, for example, a laminated film madeof SiO or SiN as illustrated in the left part of FIG. 17 is providedunder RGB color filters FR, FG, and FB as illustrated in the right partof FIG. 17 , so that the sensor that detects RGB signal components doesnot detect NIR, and the IR cut filter F is not provided only under ablack (visible light cut) filter FA of an NIR sensor. Note that theright part of FIG. 17 is a perspective appearance diagram of two pixelsx two pixels of the sensor 31, and the left part of the drawing is anenlarged cross-sectional diagram of the IR cut filter F, which showsthat infrared light IR is blocked by the IR cut filter and only light Tother than the infrared light IR transmits the sensor. Note that theblack filter FA may be configured not to include a color filter.

5. Fourth Modification Example

In addition, the sensor 31 that detects NIR component signals may beconfigured such that the IR cut filters F are provided under RGB colorfilters FC and above sensors SC as illustrated in, for example, FIG. 18so that a sensor that detects RGB signal components does not detect NIR,and the IR cut filters F are not provided only above NIR sensors SC.Note that FIG. 18 is a cross-sectional diagram for four pixels of thesensor 31, showing a configuration of a pixel P1 for NIR, a pixel P2 forlight other than NIR, a pixel P3 for NIR, and a pixel P4 for light otherthan NIR arranged from the left of the drawing.

Note that, in the above, the example in which an NDVI image is generatedbased on RGB signal components and NIR signal components and an NDVIimage growth index obtained from the generated NDVI image is used hasbeen described, however, other growth indexes may be used as long as thegrowth indexes are obtained based on the RGB signal components and NIRsignal components. Thus, it may be configured that, instead of the NDVIimage, for example, a Simple Ratio (SR) image, a Global EnvironmentMonitoring Index (GEMI) image, a Soil Adjusted Vegetation Index (SAVI)image, an Enhanced Vegetation Index (EVI) image, a PerpendicularVegetation Index (PVI) image, a Photochemical Reflectance Index (PRI)image, a Structure Insensitive Pigment Index (SIPI) image, a PlantSensing Reflectance Index (PSRI) image, a Chlorophyll Index (CI) image,a Modified Simple Ratio (mSR) image, a Modified Normalized Difference(mND) image, a Canopy Chlorophyll Index (CCI) image, a Water Index (WI)image, a Normalized Difference Water Index (NDWI) image, a CelluloseAbsorption Index (CAI) image, a Ratio Vegetation Index (RVI) image, aKind of Vegetation Index (KVI) image, and a Difference Vegetation Index(DVI) image, which are obtained based on RGB signal components and NIRsignal components, are used and growth indexes corresponding to theimages are computed and used. Furthermore, by combining a plurality ofkinds of images obtained based on RGB signal components and NIR signalcomponents, an image growth index of the combined image may be obtainedand used.

The series of processes described above can be executed by hardware orby software. When the series of processes is executed by software, aprogram constituting the software is installed in a computerincorporated into dedicated hardware, or in, for example, ageneral-purpose personal computer that can execute various functions byinstalling various programs therein from a recording medium.

FIG. 19 illustrates an illustrative configuration example of ageneral-purpose personal computer. The personal computer has a CentralProcessing Unit (CPU) 1001 built therein. The CPU 1001 is connected toan input and output interface 1005 via a bus 1004. A Read Only Memory(ROM) 1002 and a Random Access Memory (RAM) 1003 are connected to thebus 1004.

An input unit 1006 that is configured by an input device such as akeyboard or a mouse by which a user inputs operation commands, an outputunit 1007 that outputs process operation screens or process resultimages on a display device, a storage unit 1008 that is configured by ahard disk drive in which programs and various kinds of data are stored,and a communication unit 1009 that is configured by a Local Area Network(LAN) adaptor and executes communication processes via a networkrepresented by the Internet are connected to the input and outputinterface 1005. In addition, a drive 1010 that reads and writes datafrom and to a removable medium 1011 such as a magnetic disk (including aflexible disk), an optical disc (including a Compact Disc-Read OnlyMemory (CD-ROM) or a Digital Versatile Disc (DVD)), a magneto-opticaldisc (including a Mini Disc (MD)), or a semiconductor memory isconnected thereto.

The CPU 1001 executes various processes according to programs stored inthe ROM 1002 or programs read from the removable medium 1011 such as amagnetic disk, an optical disc, a magneto-optical disc, or asemiconductor memory and installed in the storage unit 1008 and loadedon the RAM 1003 from the storage unit 1008. The RAM 1003 alsoappropriately stores data for the CPU 1001 to execute various processes.Such data may be necessary for the CPU 1001 to execute variousprocesses.

A computer configured as described above performs the series ofprocesses described above when the CPU 1001 causes, for example,programs stored in the storage unit 1008 to be loaded on the RAM 1003and executes the programs via the input and output interface 1005 andthe bus 1004.

The programs executed by the computer (the CPU 1001) can be provided bybeing recorded on the removable medium 1011, for example, a packagemedium. In addition, the programs can be provided via a wired or awireless transmission medium such as a local area network, or theInternet, or in the form of digital satellite broadcasting.

The programs can be installed in the storage unit 1008 in the computervia the input and output interface 1005 by loading the removable medium1011 in the drive 1010. In addition, the programs can be installed inthe storage unit 1008 by being received by the communication unit 1009via a wired or a wireless transmission medium. In addition, the programscan be installed in the ROM 1002 or the storage unit 1008 in advance.

Note that the programs executed by the computer may be programs by whichthe processes are performed in a time-series manner in the order asdescribed herein, or may be a program by which the processes areperformed, for example, in parallel or at timings such as when there isa call-out.

In addition, in the present specification, a system means a set of aplurality of constituent elements (such as devices or modules(components)), and it does not matter that all of the constituentelements are accommodated in the same housing. Therefore, a plurality ofdevices which are accommodated in individual housings and connected toone another via a network and one device of which a plurality of modulesare accommodated in one housing falls into a system.

Note that embodiments of the presently disclosed technology are notlimited to the embodiments described above, and can be variouslymodified in the scope not departing from the gist of the presentlydisclosed technology.

For example, the presently disclosed technology can adopt the cloudcomputing configuration in which one function is divided and shared by aplurality of devices via a network.

In addition, execution of each step described in the flowcharts abovecan be done by one device or dividedly by a plurality of devices.

Further, when a plurality of processes are included in one step,execution of the plurality of processes included in the step can be doneby one device or dividedly by a plurality of devices.

Note that the presently disclosed technology can also adopt thefollowing configurations.

(A-1) A method comprising: obtaining image information of an organismcomprising a set of optical data; calculating a growth index based onthe set of optical data; and calculating an anticipated harvest timebased on the growth index, wherein the image information comprises atleast one of: (a) visible image data obtained from an image sensor andnon-visible image data obtained from the image sensor, and (b) a set ofimage data from at least two image capture devices, wherein the at leasttwo image capture devices capture the set of image data from at leasttwo positions.

(A-2) The method of (A-1), further comprising transferring theanticipated harvest time to an outside party.

(A-3) The method of (A-2), wherein the outside party is at least one ofa retailer, a general consumer, a restaurant, and a food producer.

(A-4) The method of any one of (A-1)-(A-3), wherein the visible imagedata is generated based on a demosaiced RGB pixel signal.

(A-5) The method of any one of (A-1)-(A-4), wherein the non-visibleimage data is generated based on a demosaiced R and IR signal.

(A-6) The method of any one of (A-1)-(A-4), wherein the non-visibleimage data is generated based on a R and IR signal without demosaicing.

(A-7) The method of any one of (A-1)-(A-3), wherein the visible imagedata is generated based on a demosaiced RGB pixel signal, and whereinthe near-infrared ray image data is generated based on a demosaiced Rand IR signal.

(A-8) The method of any one of (A-1)-(A-3), wherein the visible imagedata is generated based on a demosaiced RGB pixel signal, and whereinthe near-infrared ray image data is generated based on a R and IR signalwithout demosaicing.

(A-9) The method of any one of (A-1)-(A-8), wherein the set of opticaldata is obtained using a stack type image sensor, wherein the stack typeimage sensor has a blue light sensor layer stacked on a green lightsensor layer, wherein the green light sensor layer is stacked on a redlight sensor layer, and wherein the red light sensor layer is stacked onan near-infrared ray (NIR) sensor layer.

(A-10) The method of any one of (A-1)-(A-8), wherein the set of opticaldata is obtained using an image sensor comprising a set of RGB colorfilters provided over a laminated film, wherein the laminated filmcomprises at least one of SiO and SiN, and wherein the set of RGB colorfilters comprise a FR color filter, a FG color filter, and a FB colorfilter.

(A-11) The method of any one of (A-1)-(A-8), wherein the set of opticaldata is obtained using an image sensor comprising a set of RGB colorfilters provided over a set of infrared (IR) cut filters, wherein theset of IR cut filters is provided over a set of image sensors.

(A-12) The method of any one of (A-1)-(A-11), further comprisingcalculating a parallax image data based on at least two of the imagedata from the set of image data obtained from the at least two imagecapture devices; and calculating the growth index based on the parallaximage data, wherein the at least two of the image data are captured fromat least one of: two angles and the at least two positions by the atleast two image capture devices.

(A-13) A system comprising: an image capture device, wherein at leastone of the server and the image capture device is configured to: obtainimage information of an organism comprising a set of optical data;calculate a growth index based on the set of optical data; and calculatean anticipated harvest time based on the growth index, wherein the imageinformation comprises at least one of: (a) visible image data obtainedfrom an image sensor and non-visible image data obtained from the imagesensor, and (b) a set of image data from at least two image capturedevices, wherein the at least two image capture devices capture the setof image data from at least two positions.

(A-14) The system of (A-13), further comprising a server, wherein theimage capture device is in communication with the server.

(A-15) The system of (A-14), wherein the at least one of the server andthe image capture device is further configured to transfer theanticipated harvest time to an outside party.

(A-16) The system of any one of (A-13)-(A-15), wherein the visible imagedata is generated based on a demosaiced RGB pixel signal.

(A-17) The system of any one of (A-13)-(A-16), wherein the non-visibleimage data is generated based on a demosaiced R and IR signal.

(A-18) The system of any one of (A-13)-(A-16), wherein the non-visibleimage data is generated based on a R and IR signal without demosaicing.

(A-19) The system of any one of (A-13)-(A-18), wherein the set ofoptical data is obtained using a stack type image sensor, wherein thestack type image sensor has a blue light sensor layer stacked on a greenlight sensor layer, wherein the green light sensor layer is stacked on ared light sensor layer, and wherein the red light sensor layer isstacked on an near-infrared ray sensor layer.

(A-20) The system of any one of (A-13)-(A-18), wherein the set ofoptical data is obtained using an image sensor comprising a set of RGBcolor filters provided over a laminated film, wherein the laminated filmcomprises at least one of SiO and SiN, and wherein the set of RGB colorfilters comprise a FR color filter, a FG color filter, and a FB colorfilter.

(A-21) The system of any one of (A-13)-(A-18), wherein the set ofoptical data is obtained using an image sensor comprising a set of RGBcolor filters provided over a set of infrared (IR) cut filters, whereinthe set of IR cut filters is provided over a set of image sensors.

(A-22) The system of any one of (A-13)-(A-21), wherein the at least oneof the server and the image capture device is further configured to:calculate a parallax image data based on at least two of the image datafrom the set of image data obtained from the at least two image capturedevices; and calculate the growth index based on the parallax imagedata, wherein the at least two of the image data are captured from atleast one of: two angles and the at least two positions by the at leasttwo image capture devices.

(A-23) A tangible, non-transitory computer-readable medium having storedthereon instructions that cause a processor to execute a method, themethod comprising: obtaining image information of an organism comprisinga set of optical data; calculating a growth index based on the set ofoptical data; and calculating an anticipated harvest time based on thegrowth index, wherein the image information comprises at least one of:(a) visible image data obtained from an image sensor and non-visibleimage data obtained from the image sensor, and (b) a set of image datafrom at least two image capture devices, wherein the at least two imagecapture devices capture the set of image data from at least twopositions.

(A-24) The computer-readable medium of (A-23), wherein the methodfurther comprises transferring the anticipated harvest time to anoutside party.

(A-25) The computer-readable medium of (A-24), wherein the outside partyis at least one of a retailer, a general consumer, a restaurant, and afood producer.

(A-26) The computer-readable medium of any one of (A-23)-(A-25), whereinthe visible image data is generated based on a demosaiced RGB pixelsignal.

(A-27) The computer-readable medium of any one of (A-23)-(A-26), whereinthe non-visible image data is generated based on a demosaiced R and IRsignal.

(A-28) The computer-readable medium of any one of (A-23)-(A-27), whereinthe method further comprises calculating a parallax image data based onat least two of the image data from the set of image data obtained fromthe at least two image capture devices; and calculating the growth indexbased on the parallax image data, wherein the at least two of the imagedata are captured from at least one of: two angles and the at least twopositions by the at least two image capture devices.

Moreover, note that the presently disclosed technology can also adoptthe following configurations.

(B-1) An information processing system which includes an imaging unitthat captures an image of agricultural produce as an RGB image and anear-infrared ray (NIR) image, a specification unit that specifies aregion of the image in which a subject that is the agricultural produceis imaged, and a growth index computation unit that computes a growthindex of the agricultural produce based on a growth index image obtainedfrom the RGB image, the NIR image, and a red image of the RGB image ofthe region in the image which is specified by the specification unit andin which the subject is imaged.

(B-2) The information processing system described in (B-1) above, inwhich the growth index image is any one of or a combination of aNormalized Difference Vegetation Index (NDVI) image, a Simple Ratio (SR)image, a Global Environment Monitoring Index (GEMI) image, a SoilAdjusted Vegetation Index (SAVI) image, an Enhanced Vegetation Index(EVI) image, a Perpendicular Vegetation Index (PVI) image, aPhotochemical Reflectance Index (PRI) image, a Structure InsensitivePigment Index (SIPI) image, a Plant Sensing Reflectance Index (PSRI)image, a Chlorophyll Index (CI) image, a Modified Simple Ratio (mSR)image, a Modified Normalized Difference (mND) image, a CanopyChlorophyll Index (CCI) image, a Water Index (WI) image, a NormalizedDifference Water Index (NDWI) image, a Cellulose Absorption Index (CAI)image, a Ratio Vegetation Index (RVI) image, a Kind of Vegetation Index(KVI) image, and a Difference Vegetation Index (DVI) image.

(B-3) The information processing system described in (B-1) above, inwhich the imaging unit is configured to include image sensors of eachcolor of the RGB image and an image sensor for NIR.

(B-4) The information processing system described in (B-3) above, inwhich the imaging unit has a planar array of pixels having colors forthe RGB image and NIR.

(B-5) The information processing system described in (B-3) above, inwhich the imaging unit has pixels having colors for the RGB image andNIR which are arrayed so as to be laminated in a light travelingdirection.

(B-6) The information processing system described in (B-1) above, whichfurther includes a growth index image computation unit that computes thegrowth index image of the agricultural produce based on the red imageand the NIR image of the region in the image which is specified by thespecification unit and in which the subject that is the agriculturalproduce is imaged, and in which the growth index computation unitcomputes a growth index of the agricultural produce based on the growthindex image computed by the growth index image computation unit.

(B-7) The information processing system described in (B-6) above, inwhich the growth index image computation unit computes the growth indeximage from the reflectance of the near-infrared rays obtained based onthe red image and the NIR image of the region in the image which isspecified by the specification unit and in which the subject that is theagricultural produce is imaged, and a growth index of the agriculturalproduce is computed based on the average, the variance, or thehigh-order variance of the growth index image.

(B-8) The information processing system described in (B-1) above, whichfurther includes an RGB image growth index computation unit thatcomputes an RGB image growth index of the agricultural produce based onthe RGB images of the region in the image which is specified by thespecification unit and in which the subject that is the agriculturalproduce is imaged, and in which the growth index computation unitcomputes a growth index of the agricultural produce based on the RGBimage growth index computed by the RGB image growth index computationunit.

(B-9) The information processing system described in (B-8) above, inwhich the RGB image growth index computation unit computes an RGB imagegrowth index from a ratio of a predetermined color in the RGB image ofthe region in the image which is specified by the specification unit andin which the subject that is the agricultural produce is imaged.

(B-10) The information processing system described in (B-1) above, whichfurther includes a parallax image growth index computation unit thatcomputes a parallax image growth index based on a parallax imageobtained from at least two images obtained by capturing the same subjectthat is the agricultural produce from different angles, which are RGBimages of the region in the image which is specified by thespecification unit and in which the subject that is the agriculturalproduce is imaged, and in which the growth index computation unitcomputes a growth index of the agricultural produce based on theparallax image growth index computed by the parallax image growth indexcomputation unit.

(B-11) The information processing system described in (B-10) above, inwhich the parallax image growth index computation unit computes theparallax image growth index from the size of the agricultural producewhich is estimated based on the distance to the agricultural produce inan imaging direction, the size being computed based on the parallaximage obtained from at least two images obtained by capturing the samesubject that is the agricultural produce from different angles, whichare RGB images of the region in the image which is specified by thespecification unit and in which the subject that is the agriculturalproduce is imaged.

(B-12) The information processing system described in (B-1) above, whichfurther includes a storage unit that stores a position of the imagingunit, an image captured by the imaging unit, a capturing date and timeof the image captured by the imaging unit, and a growth index of eachagricultural produce captured by the imaging unit as managementinformation in association with information for identifying the imagingunit, and a harvest time computation unit that computes an anticipatedproper harvest time of the agricultural produce based on the growthindex of each agricultural produce stored in the storage unit and therelationship of a growth index and a harvest time of each agriculturalproduce of the past, and in which the storage unit also storesinformation of the anticipated proper harvest time computed by theharvest time computation unit in association with the information foridentifying the imaging unit.

(B-13) The information processing system described in (B-12) above, inwhich a sensor provided with the imaging unit, a server that manages thestorage unit storing the management information, and a terminal devicethat makes an inquiry of a harvest time to the server are included, and,when an inquiry of the anticipated proper harvest time is received fromthe terminal device, the server generates response information includingthe anticipated proper harvest time based on the management informationstored in the storage unit in response to the inquiry of the anticipatedproper harvest time based on the management information stored in thestorage unit, and transmits the response information to the terminaldevice.

(B-14) An information processing method of an information processingsystem, which includes: capturing an image of a agricultural produce asan RGB image and a near-infrared ray (NIR) image; specifying a region ofthe image in which a subject that is the agricultural produce is imaged;and computing a growth index of the agricultural produce based on agrowth index image obtained from the RGB image, the NIR image, and a redimage of the RGB image of the region in the image which is specified inthe specifying and in which the subject is imaged.

(B-15) A program that causes a computer that controls an informationprocessing system to execute: capturing an image of a agriculturalproduce as an RGB image and a near-infrared ray (NIR) image; specifyinga region of the image in which a subject that is the agriculturalproduce is imaged; and computing a growth index of the agriculturalproduce based on a growth index image obtained from the RGB image, theNIR image, and a red image of the RGB image of the region in the imagewhich is specified in the specifying and in which the subject is imaged.

(B-16) An imaging device which includes an imaging unit that captures animage of a agricultural produce as an RGB image and a near-infrared ray(NIR) image, a specification unit that specifies a region in the imagein which a subject that is the agricultural produce is imaged, and agrowth index computation unit that computes a growth index of theagricultural produce based on a growth index image obtained from the RGBimage, the NIR image, and a red image of the RGB image of the region inthe image which is specified by the specification unit and in which thesubject is imaged.

(B-17) The imaging device described in (16) above, in which the growthindex image is any one of or a combination of a Normalized DifferenceVegetation Index (NDVI) image, a Simple Ratio (SR) image, a GlobalEnvironment Monitoring Index (GEMI) image, a Soil Adjusted VegetationIndex (SAVI) image, an Enhanced Vegetation Index (EVI) image, aPerpendicular Vegetation Index (PVI) image, a Photochemical ReflectanceIndex (PRI) image, a Structure Insensitive Pigment Index (SIPI) image, aPlant Sensing Reflectance Index (PSRI) image, a Chlorophyll Index (CI)image, a Modified Simple Ratio (mSR) image, a Modified NormalizedDifference (mND) image, a Canopy Chlorophyll Index (CCI) image, a WaterIndex (WI) image, a Normalized Difference Water Index (NDWI) image, aCellulose Absorption Index (CAI) image, a Ratio Vegetation Index (RVI)image, a Kind of Vegetation Index (KVI) image, and a DifferenceVegetation Index (DVI) image.

(B-18) The imaging device described in (16) above, in which the imagingunit is configured to include image sensors of each color of the RGBimage and an image sensor for a near-infrared ray.

(B-19) An imaging method which includes: capturing an image of aagricultural produce as an RGB image and a near-infrared ray (NIR)image; specifying a region in the image in which a subject that is theagricultural produce is imaged; and computing a growth index of theagricultural produce based on a growth index image obtained from the RGBimage, the NIR image, and a red image of the RGB image of the region inthe image which is specified in the specifying and in which the subjectis imaged.

(B-20) A program that causes a computer that controls an imaging deviceto execute: capturing an image of a agricultural produce as an RGB imageand a near-infrared ray (NIR) image; specifying a region in the image inwhich a subject that is the agricultural produce is imaged; andcomputing a growth index of the agricultural produce based on a growthindex image obtained from the RGB image, the NIR image, and a red imageof the RGB image of the region in the image which is specified in thespecifying and in which the subject is imaged.

As used herein, “at least one”, “one or more”, and “and/or” areopen-ended expressions that are both conjunctive and disjunctive inoperation. For example, each of the expressions “at least one of A, Band C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “oneor more of A, B, or C” and “A, B, and/or C” means A alone, B alone, Calone, A and B together, A and C together, B and C together, or A, B andC together.

It is to be noted that the term “a” or “an” entity refers to one or moreof that entity. As such, the terms “a” (or “an”), “one or more” and “atleast one” can be used interchangeably herein. It is also to be notedthat the terms “comprising”, “including”, and “having” can be usedinterchangeably.

The terms “determine”, “calculate” and “compute,” and variationsthereof, as used herein, are used interchangeably and include any typeof methodology, process, mathematical operation or technique.

The term “computer-readable medium” as used herein refers to anytangible storage and/or transmission medium that participates inproviding instructions to a processor for execution. Such a medium maytake many forms, including but not limited to, non-volatile media,volatile media, and transmission media. Non-volatile media includes, forexample, NVRAM, or magnetic or optical disks. Volatile media includesdynamic memory, such as main memory. Common forms of computer-readablemedia include, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, magneto-optical medium, aCD-ROM, any other optical medium, punch cards, paper tape, any otherphysical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, a solid state medium like a memory card, any other memorychip or cartridge, a carrier wave as described hereinafter, or any othermedium from which a computer can read. A digital file attachment toe-mail or other self-contained information archive or set of archives isconsidered a distribution medium equivalent to a tangible storagemedium. When the computer-readable media is configured as a database, itis to be understood that the database may be any type of database, suchas relational, hierarchical, object-oriented, and/or the like.Accordingly, the invention is considered to include a tangible storagemedium or distribution medium and prior art-recognized equivalents andsuccessor media, in which the software implementations of the presentinvention are stored.

The term “module” as used herein refers to any known or later developedhardware, software, firmware, artificial intelligence, fuzzy logic, orcombination of hardware and software that is capable of performing thefunctionality associated with that element. Also, while the invention isdescribed in terms of exemplary embodiments, it should be appreciatedthat individual aspects of the invention can be separately claimed.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

REFERENCE SIGNS LIST 11, 11-1 to 11-N Sensor camera 12, 12-1 to 12-4Terminal device 13 Network 14 Server 31 Sensor 32 RGB image generationunit 33 NDVI image generation unit 34 Control unit 35 IP address storageunit 36 GPS 37 Environment information measurement unit 38 RTC 39 Growthsituation information generation unit 40 Communication unit 41Communication path specification unit 61 Control unit 62 Inquiring unit63 Operation unit 64 Communication unit 65 IP address storage unit 66Display unit 81 Control unit 82 Growth situation informationaccumulation unit 83 Target region specification unit 84 RGB imagegrowth index computation unit 85 NDVI image growth index computationunit 86 Stereoscopic image growth index computation unit 87 Harvest timecomputation unit 88 Management information accumulation unit 89 Growthindex computation unit 90 Mapping unit 91 Sensor camera operationsituation monitoring unit 92 Communication unit 93 Harvest plan creationunit 94 Delivery plan creation unit 95 Sales plan creation unit 96Purchase plan creation unit 97 Inquiry reception unit 98 Responsecreation unit

1. Information processing apparatus comprising: processing circuitryconfigured to: acquire image information of an organism comprising a setof image data; calculate a growth index based on the set of image data;calculate an anticipated harvest time based on the growth index, whereinthe set of image data is obtained from at least two image capturedevices, wherein the at least two image capture devices capture the setof image data from at least two different positions; and calculate asize of the organism based on the set of image data, wherein the set ofimage data is obtained using an image sensor comprising a set of RGBcolor filters provided over a set of infrared (IR) cut filters, whereinthe set of IR cut filters is provided over a set of RGB sensors, whereinthe image sensor further comprises a black filter provided over a NIR(near infrared) sensor with no IR cut filter provided over the NIRsensor and therefore with no blocking of NIR from the NIR sensor, andwherein an image captured from the RGB sensors and the NIR sensor isrepresentative of the organism.
 2. The information processing apparatusof claim 1, wherein the image information comprises visible image dataand non-visible image data and wherein the visible image data isgenerated by demosaicing of a RGB pixel signal to provide R, G and Bcomponent image signals and wherein the non-visible image data isgenerated based on a demosaiced R and IR signal.
 3. The informationprocessing apparatus of claim 1, wherein the image information comprisesvisible image data and non-visible image data and wherein the visibleimage data is generated by demosaicing of a RGB pixel signal to provideR, G and B component image signals and wherein the non-visible imagedata is generated based on a R and IR signal without demosaicing.
 4. Theinformation processing apparatus of claim 1, wherein the image sensorused to obtain the set of image data comprises a stack type imagesensor, wherein the stack type image sensor has a blue light sensorlayer stacked on a green light sensor layer, wherein the green lightsensor layer is stacked on a red light sensor layer, and wherein the redlight sensor layer is stacked on a near-infrared ray sensor layer. 5.The information processing apparatus of claim 1, wherein the imagesensor used to obtain the set of image data comprises the set of RGBcolor filters provided over a laminated film, wherein the laminated filmcomprises at least one of SiO and SiN, and wherein the set of RGB colorfilters comprises a FR color filter, a FG color filter, and a FB colorfilter.
 6. The information processing apparatus of claim 1, wherein theprocessing circuitry is configured to calculate parallax image databased on at least two of the image data from the set of image dataobtained from the at least two image capture devices and to calculatethe growth index based on the parallax image data.
 7. Informationprocessing apparatus comprising: processing circuitry configured to:acquire image information of an organism comprising a set of image data;calculate a growth index based on the set of image data; and calculatean anticipated harvest time based on the growth index, wherein the setof image data is obtained from at least two image capture devices,wherein the at least two image capture devices capture the set of imagedata from at least two different positions, wherein the set of imagedata is obtained using an image sensor comprising a set of RGB colorfilters provided over a set of infrared (IR) cut filters, wherein theset of IR cut filters is provided over a set of RGB sensors, wherein theimage sensor further comprises a black filter provided over a NIR (nearinfrared) sensor with no IR cut filter provided over the NIR sensor andtherefore with no blocking of NIR from the NIR sensor, and wherein animage captured from the RGB sensors and the NIR sensor is representativeof the organism.
 8. The information processing apparatus of claim 7,wherein the image information comprises visible image data andnon-visible image data and wherein the visible image data is generatedby demosaicing of a RGB pixel signal to provide R, G and B componentimage signals and wherein the non-visible image data is generated basedon a demosaiced R and IR signal.
 9. The information processing apparatusof claim 7, wherein the image information comprises visible image dataand non-visible image data and wherein the visible image data isgenerated by demosaicing of a RGB pixel signal to provide R, G and Bcomponent image signals and wherein the non-visible image data isgenerated based on a R and IR signal without demosaicing.
 10. Theinformation processing apparatus of claim 7, wherein the image sensorused to obtain the set of optical data comprises a stack type imagesensor, wherein the stack type image sensor has a blue light sensorlayer stacked on a green light sensor layer, wherein the green lightsensor layer is stacked on a red light sensor layer, and wherein the redlight sensor layer is stacked on a near-infrared ray sensor layer. 11.The information processing apparatus of claim 7, wherein the imagesensor used to obtain the set of optical data comprises the set of RGBcolor filters provided over a laminated film, wherein the laminated filmcomprises at least one of SiO and SiN, and wherein the set of RGB colorfilters comprises a FR color filter, a FG color filter, and a FB colorfilter.
 12. The information processing apparatus of claim 7, wherein theprocessing circuitry is configured to calculate parallax image databased on at least two of the image data from the set of image dataobtained from the at least two image capture devices and to calculatethe growth index based on the parallax image data.
 13. An image sensorcomprising: RGB sensors; a NIR (near infrared) sensor; IR (infrared) cutfilters over the RGB sensors; RGB color filters over the IR cut filters;and a black filter over the NIR sensor with no IR cut filter over theNIR sensor and therefore with no blocking of NIR from the NIR sensor.14. The image sensor of claim 13, wherein the image sensor used toobtain the set of image data comprises a stack type image sensor,wherein the stack type image sensor has a blue light sensor layerstacked on a green light sensor layer, wherein the green light sensorlayer is stacked on a red light sensor layer, and wherein the red lightsensor layer is stacked on a near-infrared ray sensor layer.
 15. Theimage sensor of claim 13, wherein the image sensor used to obtain theset of image data comprises the set of RGB color filters provided over alaminated film, wherein the laminated film comprises at least one of SiOand SiN, and wherein the set of RGB color filters comprises a FR colorfilter, a FG color filter, and a FB color filter.